Using Linguistic Information to Improve the Performance of Vector-Based Semantic Analysis

In this paper, we will show that the performance of vector-based semantic analysis can be improved by considering basic linguistic structures in the data— e.g. morphology. For this purpose, we have used a new method for vector-based semantic analysis that computes semantic word vectors based on distributed representations by means of random labeling of words in narrow context windows. This form of representation is more natural than previously reported techniques, and, as we will show, equivalent or even superior in performance when subjected to a standardized synonym test. The use of vector-based models of information for the purpose of semantic analysis is an area of research that has gained substantial recognition over the last decade. have demonstrated the viability of computing semantic word vectors from the co-occurrence statistics of words in large text data. However, the prevailing techniques have been almost exclusively statistical, and consequently paid little or no attention to the linguistic structures of the data used in the experiments. This negligence regarding linguistics has, of course, been at least partly deliberate, as one of the primary goals of the techniques has been to develop representations of word meanings from text data " that was minimally preprocessed, not unlike human human-concept acquisition " (Burgess & Lund, 1998). LSA and HAL are both purely statistical methods that treat the text data simply as a bag-of-words in which the only relevant piece of structural information is the words-by-contexts co-occurrence frequencies. What separates the two approaches is their treatment and conception of. In LSA, the text data is represented as a words-by-documents co-occurrence matrix where each cell indicates the frequency of a given word in a given text sample of approximately 150 words. The frequencies are normalized, and the normalized matrix is transformed with Singular Value Decomposition (SVD) into a smaller matrix with reduced dimensionality. The purpose of using SVD to reduce the dimensions of the normalized frequency matrix is that this operation appears to accomplish inductive effects that capture latent semantic structures in the text data. Words are thus represented in the reduced matrix by semantic vectors of £ dimensionality (300 proving to be optimal in Landauer & Dumais' (1997) experiments). In HAL, the data is represented as a words-bywords co-occurrence matrix where each cell indicates the co-occurrence counts for a single word pair (a word pair being an asymmetrical relation so that " ¤ ¢ ¥ " and " ¦ …